Computerized devices control almost every aspect of our life—from writing documents to controlling traffic lights. However, computerized devices are bug-prone, and thus require a testing phase in which the bugs should be discovered. The testing phase is considered one of the most difficult tasks in designing a computerized device. The cost of not discovering a bug may be enormous, as the consequences of the bug may be disastrous. For example, a bug may cause the injury of a person relying on the designated behavior of the computerized device. Additionally, a bug in hardware or firmware may be expensive to fix, as patching it requires call-back of the computerized device. Hence, many developers of computerized devices invest a substantial portion of the development cycle to discover erroneous behaviors of the computerized device.
During testing phase, developers, QA staff members, and the like test application to verify that a newly developed application operates properly. In some cases, test cases (also referred to simply as “tests”) may be devised to provide stimuli to the application, and enable testing whether its operation is correct (e.g., as expected).
However, as the total number of possible stimuli is usually large, testing can only falsify the requirements (i.e., exhibit execution in which the requirements are not met), but cannot verify correctness (i.e., prove that every execution conforms to the requirements). A measurement may be used to determine whether sufficient testing of the application was performed. Coverage analysis provides one such measurement.
A coverage model defines coverage tasks. A coverage measurement may be measured by comparing the number of covered coverage tasks with the total number of coverage tasks defined by the coverage model. For example, if the coverage model defines coverage tasks for each function of the application, and a function that is invoked during execution is considered as covering the correlating coverage task, then the greater the number of different functions that are invoked during the entire testing process, the higher the coverage measurement.
Various coverage models exist. Each defines different coverage tasks. A code coverage model defines coverage tasks associated with code elements of the Software-Under-Test (SUT). Various types of code coverage models exist, such as: function coverage (a coverage task is the execution of a function), statement coverage (a coverage task is the execution of a statement), branch coverage (a coverage task is traversing a control flow edge (i.e., taking a “branch”)), decision coverage (a coverage task is each control structure in the SUT (such as an if statement); each control structure should be evaluated both to true and false), condition coverage, also known as predicate coverage (a coverage task is each Boolean sub-expression used by the SUT; each Boolean expression should be evaluated both to true and false), condition/decision coverage (a combination of condition coverage and decision coverage), modified condition/decision coverage (MC/DC) (adds to the condition/decision coverage metric the requirement that each condition should affect the decision outcome independently), path coverage (a coverage task is every possible route through the code of the SUT), Exit/entry coverage (a coverage task is every possible invocation and return of each function), and the like.
After coverage is measured, review and analysis of the testing phase may be performed to improve it based on the measured coverage. One example would be to determine whether “sufficient” testing was performed. In some cases, should the coverage measurement by above a predetermined threshold, it may be concluded that “sufficient” testing was performed. Another example would be to design a test that would cover an uncovered coverage tasks. Yet another example would be to perform test selection: selection of a subset of executed tests that would induce the same coverage measurement. Test selection may enable future testing to be more efficient by reducing duplicity. Yet another example would be to group several related coverage tasks (some of which may be covered by some test suits while other may not) into sets, and perform coverage analysis of groups of tasks thus reducing the total number of items the user needs to review.